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Introduction on CBQ 


It is a link sharing mechanism which tries to provide real-time 
service. 


It consists of classes, each associated with certain share of 
bandwidth and a priority. 


It consists of a root class, interior classes and leaf classes defined 


in a hierarchy. Only leaf classes have physical queues. 


A flow or group of flows are assigned to a class. 


It provides isolation among classes of traffic while allows them to 


share the bandwidth of the link. 


Each class should receive roughly its allocated bandwidth over some 
interval of time, even during congestion. 
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Introduction on CBQ 


Link-sharing provides a means for sharing the bandwidth of a link 
among different organizations, protocols, etc. 


When some classes are not using their allocated bandwidth, the 
distribution of the excess bandwidth among the other classes follows 
some appropriate set of guidelines. 


Classes with higher priority are served first. Therefore, the packets 
that belong to these classes generally experience lower queueing delay. 


Priority-based scheduling can be used to reduce delay for the real-time 
traffic. 


Priority-based scheduling alone causes starvation. Link-sharing 
mechanism needed to serve the classes in accordance to their allocated 


bandwidth and prevent certain classes from hijacking the bandwidth. 


Introduction on CBQ 


e General scheduler schedules packets from leaf classes without regard to 
link-sharing guidelines. 


e General scheduler:- 


— Packet round-robin 


— Weighted round-robin 


e Link-sharing schedules packets from some leaf classes that have been 
exceeding their link-sharing allocation in times of congestion. 
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Introduction on CBQ 


e Link-sharing guidelines:- 


— Formal link-sharing guideline (i) The class is not overlimit OR 
(ii) The class has a not-overlimit ancestor at level i, and there are no 


unsatisfied class in the link-sharing structure at levels lower than 2. 


— Ancestor-only guideline (i) The class is not overlimit OR (ii) the 
class has an underlimit ancestor. 


— Top-level guideline (i) The class is not overlimit OR (ii) the class 
has an underlimit ancestor whose level is at most ‘Top-Level. 


Link-Sharing Mechanism of CBQ 
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Priority 2 


Class N 


Components of CBQ 


Implemented at outgoing interface. 


Packet classifier maps packets arriving at the gateway to the 
appropriate classes for that output link. 


Link-sharing framework maintains link-sharing constraints. 


Packet Scheduler schedules classes according to their bandwidth 
allocation and priority. 


Estimator estimates the bandwidth used by each class over an 
appropriate time interval. 


Traffic control interface provides a means for setting, and 
configuring classes, creating filters at classifier, and collection of 
statistics. 
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Components of CBQ 


e Network device is an abstraction layer that hides the complexity and 
differences of device drivers from the kernel. 


| 
i] y y 


User Space 


A A A 
A A | 


Kernel Space 


Linux Implementation of CBhQ 


A port of ALTQ from FreeBSD. 


ALTQ provides a framework that allows other packet schedulers to be 
implemented in the kernel. 


A wrapper for ALTQ is implemented in altq.c. It provides functions 


for activating and disabling certain packet scheduler; extracting flow 


information; and interface to device driver of packet schedulers. 


ALTQ and other packet schedulers are controlled through a character 
device (e.g. /dev/altq and /dev/cbq). Open, close and ioctl are used 
by user space program to control ALTQ and packet schedulers. 


dev.c and device data structure are modified to divert packets to 
ALTQ module. 


March 5-7, 1999 Singapore Linux Conference 12 


Device Data Structure 


struct device { 


struct if_statistic* (*get_wireless_stats) (struct device *dev); 


#ifdef CONFIG_ALTQ 


int if_altqflags; /* altq flags */ 
void *if_altqp; /* pointer to altq state data */ 


int (*if_altqenqueue) (struct device *, struct sk_buff *, 
struct flowinfo *, int); 
struct sk_buff* (*if_altqdequeue(struct device *, int); 
#endif 
ay 


Flow Chart 


ip_queue_xmit() ip_build_xmit() 


Y ' 


dev_queue_xmit() 


Y 


do_dev_queue_xmit() 


is ALTQ 
active? 
altq_extractflow() 


Y 


if_altqenqueue() __skb_queue_tail() 


y Y 
if_altqdequeue() __skb_dequeue() 


pass packet to 
network card 
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Linux Implementation of CBQ 


Interface to CBQ is implemented in cbq.c. Provides functions for 
creating and removing classes as well as filters; getting statistics; 
enqueueing and dequeueing packets. 


Packet classifier is implemented in cbq_class.c. Flows are identified by 
their source address and port, together with destination address and 
port. 


rm_class.c performs the actual packet scheduling. 


rmc_dequeue_next performs WRR on classes of the same priority. 
Each class is assigned a weight that is proportional to the its 


bandwidth allocation. The weight determines the number of bytes that 


a class is allowed to send each round. If a class sent more than its 
share, it will be penalized in future rounds. 


Linux Implementation CBQ 


When a class is overlimit, it is prohibited from sending any packet. 
rmc_delay_action is invoked to calculate the next time that the class 
is allowed to transmit again. 


Exponential Weighted Moving Average is used to estimate the 


bandwidth consumption of each class. This estimating function is 


implemented in rmc_update_util. 
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Transmission of Packet 
The information of the packet is extracted by altq_extractflow. 
Packet classified by cbq_classify. 
Packet inserted into the appropriate queue by rmc_queue_pkt. 


Check if there is any packet ready for transmission 
(rmc_dequeue_next). 


Update the bandwidth usage (rmc_update_util). 


do_dev_queue_xmit altq_extractflow if_altqenqueue cbhq_classify 
rmc_update_util rmc_dequeue_next cbq_dequeue if_altqdequeue rmc_queue_pkt 


Packet is dropped 
if buffer is full 
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Linux Implementation of CBhQ 


e ‘Timer interrupt is invoked every 20ms. 


e Call cbq_watchdog which in turn calls dev_tint. 


e dev_tint modified to check if there is packet to transmit. 


timer interrupt cbq_watchdog dev_tint do_dev_queue_xmit 


A 


asia a oe ee 


Called by when an 
interface is ready to 
transmit packet 
Additional code here 
' to kick start transmission | 
of packet. 
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Sample Content of Proc File 


e A proc file is implemented for CBQ. 
e Allows easy access to the status of CBQ from user space. 


e Contains statistics of each class. 


| dropped | 
cls prty| pkts bytes | ns_byte qcnt 
0 0 0 833 0 


3 2 fe) 1176 te) 
6 te) 6) 40000 0 
6 0 6) 2666 0 


sample Config File 


e Create 2 classes:— res_class and unres_class. 


e res_class:- 


50% of bandwidth, priority 6, used by RSVP to make 


reservation. 


e unres_class:- 50% of bandwidth, priority 3, used by best-effort traffic. 


interface eth0O 
class cbq eth0O 
class cbq eth0O 
borrow 
class cbq eth0d 
borrow 
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bandwidth 10000000 cbq 

root_class null priority O admission none pbandwidth 100 
res_class root_class priority 6 pbandwidth 50 admission cntlload 
root_class 

unres_class root_class priority 3 pbandwidth 50 default 
root_class 
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cbqd — Simple CBQ daemon 


e Can be run in background or interactive mode. 


Enter ? or command: 


cbq > ? 

DoCommand: ? 
Commands are: 
help | ? 
quit 


interface if_name [bandwidth bps] [sched_type] 


class cbq if_name class_name parent [borrow borrow_class] 


[admission ctlload|none] [maxburst count] [minburst count] 


[packetsize bytes] [priority pri] [pbandwidth percent] 


filter if_name class_name dst [netmask #] dport src [netmask #] sport proto 


cbq if_name {enable|disablelacc enablelacc disable} 
cbq > 
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Implementation Issues 


Allow choice between FIFO and CBQ. 


New qucueing discipline can be added to the ALTQ framework by 


adding a new entry in the ALTQ device table. 


Interrupts must be blocked during dequeue process to prevent race 
condition. 


The completion time of packet is estimated rather than using a 
callback method. 


A timer is used to periodically check whether regulated class has been 
allowed to transmit again. 
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Implementation Issues 


e Timer granularity. Packet completion time calculated based on 
do_gettimeofday. 


e An overlimit class is suspended by CBQ. Resumption of transmission is 
triggered by either a send/receive event, or a timer interrupt which has 
granularity of 20ms. In the worst case, the timing will be rounded up 


by at least 20ms. 


Network Setup for Testing CBQ at 
End-hosts 


10Base-2 


Source 


Pentium 90MHz Pentium 90MHz 
24 Mbytes RAM 24 Mbytes RAM 
Etherlink Ill Combo ISA Etherlink Ill Combo ISA 
(3C509B) (8C509B) 
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Classes for Testing Overhead of CBQ 


default control 


Effect of CBQ on Throughput 


e TCP 


Request size (bytes) | Throughput (Mbps) 
(CBQ) 
128 8.41 


Throughput (Mbps) 
(no CBQ) 
8.40 


256 8.38 
512 8.43 
1024 8.41 


Packet size (bytes) | Throughput (Mbps) 
(CBQ) 


8.40 
8.40 
8.41 


Throughput (Mbps) 
(no CBQ) 
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Latency Caused by CBQ 


Request Response with CBQ 


no CBQ | Latency 


size (bytes) | size (bytes) | RTT (usec) | RIT (psec) (usec) 


1 625.9 


597.4 28.5 


26 


Latency Incurred by 


2500 T T T T T T T T 


5 6 
Number of Flows 
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UDP TCP 


Network Setup for Testing CBQ at 
End-hosts and Router 


Subnet X.X.X.160 Subnet: X.X.X.176 
(Netmask: 255.255.255.240) (Netmask: 255.255.255.240) 


Source A = Router L Sink 


10Base-2 10Base-2 


(Link A) (Link B) 
A A 


Interface A Interface B Interface C Interface D 
X.X.X162 X.X.X.161 X.X.X.177 X.X.X.178 


Pentium 90 Pentium 90 Pentium 90 
24M RAM 24M RAM 32M RAM 
ErtherLink Il Combo (ISA) ErtherLink II Combo (ISA) ErtherLink III Combo (ISA) 
3C509B 3C509B 3C509B 
Linux 2.0.30 Linux 2.0.30 Linux 2.0.30 
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Classes for Multiple UDP Flows 


default 


class 1 class 2 class 3 class 4 
10% 20% 30% 40% 


30 
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Bandwidth Guarantee for UDP Traffic 


Timing Diagram 


Class 4 


Class 3 


Class 2 


Class 1 


Default 


5 10 15 20 25 30 35 40 45 50 55 
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Classes for Multiple TCP Flows 


default 


class 1 class 2 class 3 class 4 
20% 20% 10% 10% 
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Bandwidth Guarantee for TCP 
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Throughput of UDP Flows 


Default — 
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Default — + 
Class 1 ---- 
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Default — 4 F \ Default — 4 
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Throughput(Mbps) 
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CBQ Enabled CBQ Disabled 
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Total Packets Dropped 


T T T T T 
Total Packets dropped (Default) — Total Packets dropped (Default) — 
Total Packets dropped (Class 1) ---- Total Packets dropped (Class 1) ---- 


8 5 6 
Time (s) Time (s) 


CBQ Enabled CBQ Disabled 
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Delay Jitter 
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Arrival Times 


Packet Sequence Number 


Default --- +4 
Class 1 -+-- 


64 6.6 6.4 6.6 
Arrival Time (s) Arrival Time (s) 


CBQ Enabled CBQ Disabled 
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Transmission Delay 


Default --— 
Class 1 -+-- 
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Default --— 
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Delay Jitter (with CBQ) 
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Conclusion 


Able to provide bandwidth guarantee. 
Provides lower delay to higher priority queues. 
Lower packet loss for higher priority queue. 
Required at end-hosts and routers. 


Does not scale with number of flows. 


Could introduces delay jitter if not properly configured. 
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